Quantization reduces the precision of input values. The lower precision outputs can be represented with fewer bits, thereby achieving data compression. Quantization is an important step in all video coding standards.
FIG. 1 illustrates the quantization process 100. Quantization divides an integer input F by an integer quantization step Q. The quotient is an integer and a fractional part. The resultant A includes rounding the fractional part by a method based on the coding standard. The rounding conventions for several standards are shown in Table 1.
TABLE 1Video Coding StandardRounding ConventionMPEG-1/2 Intra DCTRound to the nearestcoefficientsintegerMPEG-1/2 Inter DCTRound with truncationcoefficientstoward zeroH.263 (MPEG-4 type 2)Round with truncationIntra AC and Intertoward zeroDCT coefficientsFIG. 2a illustrates the relationship between the input F, the quantization step Q and the output A using the truncation toward zero (A=trunc(F/Q)). FIG. 2b illustrates the quantization error tr_err(F/Q). The quantization error tr_err(F/Q)=F/Q−trunc(F/Q).
Division takes a lot of computation compared to addition, subtraction and multiplication. Division is more complicated than these other operations when embodied in hardware circuits. The amount of computation is important in video coding, because a lot of data must be quantized. Thus quantization is often achieved indirectly. A common method replaces division by multiplication between the numerator and the reciprocal of the denominator. Moreover, the reciprocal of denominator is not represented as a floating point number but as a fixed-point number. In fixed-point representation the decimal point is implicitly placed between bits of the binary representation of a number. The decimal point position is selected depending on the required precision and range. With fixed-point representation, all arithmetic operations use integer arithmetic. This improves computation speed and reduces complexity.
Table 2 lists definitions of some functions used in this application for reference.
TABLE 2TermDefinitiontrunc(x)Rounding with truncationtoward zeroround(x)Rounding to nearest integertr_err(x)Truncation error functionx - trunc (x)rd_err(x)Rounding error functionx - round (x)
The program code listing below shows a common fixed-point implementation of the truncation quantizer trunc(F/Q). This implements integer division with truncation towards zero.
/* Fixed-point Implementation of Quantizer trunc(T/Q)*//* F, q1 and a are fixed point numbers*/unsigned int F;/* 0 bits after decimal point*/unsigned int q1;/* Fixed point representation of  1/Q, i.e. q1 = 1/Q, r bits  after decimal point*/unsigned int a;F = abs (T);a = F * q1;/* a 0+r bits after decimal point*/A2 = a >> r/* shift a by 0+r bits, obtains  integer part only of quotient*/return A2 * sign(T);This code listing is valid for any integer T and positive integer Q. Table 3 shows a comparison between direct division and this fixed-point algorithm.
TABLE 3Fixed PointDirect DivisionImplementationtrunc(F/Q)(F − q1) >> rCPU cycles559per inputThe fixed-point implementation is considerably faster than direct division. This fixed-point implementation has been used in many products and video end-equipment. This fixed-point implementation of quantizer will be used as the baseline quantizer in this application.